VTD-XML
Virtual Token Descriptor for eXtensible Markup Language (VTD-XML) refers to a collection of efficient XML processing technologies centered around a non-extractiveNon-extractive Parsing for XML XML, "document-centric" parsing technique called Virtual Token Descriptor (VTD). Depending on the perspective, VTD-XML can be viewed as one of the following: *A "Document-Centric" Manipulate XML Content the Ximple Way Introduce the concept of document centric XML Processing and why it is where the future liesVTD-XML: XML Processing for the Future (Part II)XML parserSimplify XML Processing with VTD-XMLBetter Faster XML Processing with VTD-XML VTD-XML: XML Processing for the future (Part I) *A native XML indexer or a file format that uses binary data to enhance the text XML Index XML documents with VTD-XML *An incremental XML content modifier XML content update using XMLModifier (Part I)XML content update using XMLModifier (Part II) *An XML slicer/splitter/assembler Cut, Paste, Split and Assemble XML documents with VTD-XML *An XML editor/eraser *A way to port XML processing on chipXML on a chip?XML on a ChipXimpleWare's W3C binary XML workshop Position Paper *A non-blocking, stateless XPath evaluator Improve XPath efficiency with VTD-XML VTD-XML is developed by XimpleWare and dual-licensed under GPL and proprietary license. It is originally written in Java, but is now available in CVTD-XML: a New Vision of XML by courtesy of Victor R. Volkman and C#. Basic Concept Non-Extractive,Document-Centric Parsing Traditionally, a lexical analyzer typically represents tokens (the small units of indivisible character values) as discrete string objects. This approach is known as "extractive" parsing. In contrast, "non-extractive" tokenization mandates that one keep the source text intact, and use offsets and lengths to describe those tokens. Virtual Token Descriptor Virtual Token Descriptor (VTD) applies the concept of non-extractive, document-centric parsing to XML processing. A VTD record uses a 64-bit integer to encode the offset, length, token type and nesting depth of a token in an XML document. Because all VTD records are 64-bit in length, they can be efficiently stored and managed as an array. Location Cache Location Caches (LC) build on top of VTD records to provide efficient random access. Organized as tables on a per-depth-level basis, LC entries model an XML document hierarchy consisting entirely of elements. An LC entry is a 64-bit integer encoding a pair of 32-bit values: the upper 32 bits point to the VTD record for the corresponding element, and the lower 32 bits point to the first child index value in the LC table one nesting level below. Benefits Overview Virtually all the core benefits of VTD-XML are inherent to non-extractive, document-centric parsing whose key characteristics include: *The source XML text is kept intact in memory and undecoded. *The internal representation of VTD-XML is inherently persistent. *VTD-XML abolishes object-oriented modeling of the hierarchical representation as it relies entirely on primitive data types (ie. 64-bit integers) to represent the XML hierarchy, thus reducing object creation cost to nearly zeroi-Technology Viewpoint: The Performance Woe of Binary XML @ SOA WORLD MAGAZINE. Combining those characteristics opens the door to thinking XML purely as syntax (bits,bytes, offsets,lengths, fragments, namespace-compensated fragments, and document composition) instead of serialization/deserialization of objects. This new thinking can become an increasingly powerful way to think about XML/SOA applications. As Parser When used in parsing mode, VTD-XML is a general purpose, high-performance VTD-XML Parsing/Navigation Perforamnce ReportXML parser. The key technical characteristics are: * Performance-wise, VTD-XML typically outperforms SAX (with NULL content handler) by 100%, while still providing full random access and built-in XPath support. * In terms of memory usage, VTD-XML typically consumes memory 30%~50% greater than the XML document size, which is around 1/5 the memory usage of DOM * Applications written in VTD-XML are usually much shorter and cleaner than their DOM or SAX versions. As Indexer Because of the inherent persistence of VTD-XML, developers can choose to write the internal representation of a parsed XML document on disk, and later load it back in memory to avoid repetitive parsing. To this end, XimpleWare has introduced VTD+XML as a binary packaging format combining VTD, LC and the XML text. It can typically be viewed in one of the two following ways: *A native XML index that completely eliminates the parsing cost and also retains 100% benefits of XML. It is a file format that is human readable and backward compatible with XML. *A binary XML format that uses binary data to enhance the processing of the XML text. XML Content Modifier Because the XML text is kept intact and undecoded by VTD-XML, when an application intends to modify the content of XML, it only needs to modify the portions of XML content most relevant to the changes. This is in stark contrast with DOM, SAX or StAx parsing, which incur the cost of parsing and re-serialization no matter how small the changes are. However, since VTDs refer to document elements by their offsets, changes to the length of elements occurring earlier in a document will require adjustments to VTDs that refer to later elements. XML Slicer/Splitter/Assembler An application based on VTD-XML can also use offsets and lengths to address tokens, or element fragments. This allow XML documents to be manipulated like arrays of bytes. *As a slicer, VTD-XML can "slice" off a token or an element fragment from an XML document, then insert it back into another location in the same document, or into a different document. *As a splitter, VTD-XML can split sub-elements in an XML document and dump each into a separate XML document. *As an assembler, VTD-XML can "cut" chunks out of multiple XML documents and assemble them into a new XML document. XML Editor/Eraser Used as an editor/eraser, VTD-XML can directly edit/erase the underlying byte content of the XML text, provided that the token length is wider than the intended new content. An immediate benefit of this approach is that the application can immediately reuse the original VTD and LC. In contrast, when using VTD-XML to incrementally update an XML document, an application needs to reparse the updated document before the application can process it. Other Benefits VTD-XML also pioneers the non-blocking, stateless XPath evaluation approach. Weaknesses VTD-XML also exhibits a few noticeable shortcomings: *As an XML parser, it does not support external entities declared in DTD. *As a file format, it increases the document size by about 30% to 50%. *As an API, it is not compatible with DOM or SAX. Furthermore, in VTD-XML it is difficult to support certain validation techniques (e.g. default attribute and elements) employed by DTD and XML Schema that require modifications to the XML instances being assessed. Areas of Applications General-purpose Replacement for DOM or SAX Because of VTD-XML's performance and memory advantages, it covers a larger portion of XML use cases than either DOM or SAXhttp://www.devx.com/xml/Article/30484 Process Large XML documents with VTD-XML. *Compared to DOM, VTD-XML processes bigger (3x~5x) XML documents for the same amount of physical memory at about 3 to 10 times the performance. *Compared to SAX, VTD-XML provides random access and XPath support and outperforms SAX by at least 2x. For SOA/WS/XML Security The combination of VTD-XML's high performance and incremental-update capability makes it essentialAccelerate WSS Applications with VTD-XMLW3C workshop presentation on XML security http://www.w3.org/2007/xmlsec/ws/papers/06-zhang-ximpleware/to achieve the desired level of QoS for those applications. For SOA/WS/XML Intermediary VTD-XML is well suited for SOA intermediary applications such as XML routers/switches/gateways, Enterprise Service Buses, and services aggregation points. All those applications perform the basic "store and forward" operations for which retaining XML as-is is critical for minimizing latency. VTD-XML's incremental update capability also contributes significantly to the fowarding performance. VTD-XML's random-access capability lends itself well to XPath-based XML routing/switching/filtering common in AJAX and SOA deployment. Intelligent SOA/WS/XML Load-balancing and Offloading When an XML document travels through several middle-tier SOA components, the first message stop, after finishing inspecting the XML document, can choose to send the VTD+XML file format to the downstream components to avoid repetitive parsing, further improving the performance of the message flow. By the same token, an intelligent SOA load balancer can choose to generate VTD+XML for incoming/outgoing SOAP messages to offload XML parsing of the application servers at the receiving end of those messages. XML Persistence Data Store When viewed from the perspective of native XML persistence, VTD-XML can be used as a human-readable, easy to use, and general-purpose XML index. XML documents stored this way can be loaded into memory to be queried, updated or edited without the overhead of parsing/re-serialization. Schemaless XML Data Binding VTD-XML's combination of high performance, low memory usage and non-blocking XPath evaluation makes possible a new XML data binding approach based entirely on XPath. This approach's biggest benefit is it no longer requires XML schema, avoids needless object creation, and takes advantage of XML's inherent loose encoding Schemaless Java-XML data binding with VTD-XML. Essential Classes As of Version 2.2, the Java and C# versions of VTD-XML consist of the following classes: * VTDGen (VTD Generator) is the class that encapsulates the main parsing, index loading and index writing functions. *'VTDNav' (VTD Navigator) is the class that (1) encapsulates XML, VTD, and hierarchical info, (2) contains various navigation methods,(3) performs various comparisons between VTD records and strings, and (4) converts VTD records to primitive data types. *'AutoPilot' is a class containing functions that perform node-level iteration and XPath. *'XMLModifier' is a class that offers incremental update capability, such as delete, insert and update. Code Sample /* In this java program, we demonstrate how to use XMLModifier to incrementally * update a simple XML purchase order. * a particular name space. We also are going * to use VTDGen's parseFile to simplify programming. */ import com.ximpleware.*; public class update { public static void main(String argv[]) throws NavException, ModifyException,IOException{ // open a file and read the content into a byte array VTDGen vg = new VTDGen(); if (vg.parseFile("oldpo.xml", true)){ VTDNav vn = vg.getNav(); AutoPilot ap = new AutoPilot(vn); XMLModifier xm = new XMLModifier(vn); ap.selectXPath("/purchaseOrder/items/item@partNum='872-AA'"); int i = -1; while((i=ap.evalXPath())!=-1){ xm.remove(); xm.insertBeforeElement("\n"); } ap.selectXPath("/purchaseOrder/items/item/USPrice.<40/text()"); while((i=ap.evalXPath())!=-1){ xm.updateToken(i,"200"); } xm.output("newpo.xml"); } } } External links Category:XML Category:XML parsers Category:Free software Category:Java platform Category:SourceForge projects Category:.NET programming tools